Balancing between Speed and Rigor in Qualitative Reporting and Emerging Usage of AI Tools
Jun 26, 2024, Jiten Madia
(This article is written based on 30 to 45 minutes conversations with 5 well experienced qualitative researchers. The conversations happened in March and April. AI is a fast-developing field and some of these findings may have changed.)
Imagine being asked to read Harry Potter book 1 and 2 and create an insightful report in 1 week. And then, when you are presenting it, the client asks
“What position do the Weasley twins play in Quidditch?”
And you are wondering was it there in the data or not covered.
Well, this is not an entirely imaginary situation if you have ever presented findings of 20 Depth Interviews* to a client. The Qualitative Research process produces copious amounts of text data. Analysing this data can be a time-consuming process. And what we all know is that every client wants results/report yesterday. The demand for results can come either in form of topline, quick debriefs or a super shortened report writing window.
When as researcher you go and present your research findings, you want to ensure you can answer all reasonable client questions, and with significant ease.
So, how do researchers cope with the timeline pressure? How do they ensure their work is of reasonable quality? Balancing rigor and speed in qual research can be tough. We asked 5 well experienced, renowned, and thorough researchers. We met Dean Stephens, Janet Standen, Doug Keith, Jenny Brandt and Leonore Saintville.
The findings are super interesting.
Researchers manage the balance between speed and rigour in their qualitative research report writing process through several strategies and tools. The approach can vary depending on the project size, type, and specific client needs. Timelines play an important role, too. Sometimes, full report may not be required for decision-making. Researchers focus on ensuring that study findings do not lose their relevance by managing timelines. Exploratory studies might require more detailed analysis, while concept testing might rely more on quick, structured feedback.
Qualitative Data reporting can be made incredibly fast by clearly dividing the process into 2 parts:
- Analysis
- Report writing
There are primarily 3 ways researchers speed up the analysis process:
Operational Level Excellence – Researchers often start with thorough pre-planning, including creating detailed note-taking templates and briefing note-takers. This helps ensure that data collection is structured and aligned with the project objectives, which speeds up the later stages of analysis and report writing.
Researchers also ensure that the note-takers are very capable. Some researchers even use a fellow researcher or associate/junior researcher for note-taking purposes.
Some researchers also allow note taking to take place after the interview is completed by watching the video recording.
Use of Technology/AI - Tools like Reduct, CoLoop, and Recollective were mentioned along with generic resources like ChatGPT (paid version). Reduct is useful in doing video coding and hence useful for certain kinds of projects. Co-loop, Recollective, Yabble and other tools can help in automated transcription as well as querying the data. researchers are using AI tools to significantly reduce the time required for manual transcription and coding.
Stakeholder Involvement - When the clients cannot afford long report timelines, the research process is designed as an iterative process. Researchers conduct immediate post-session debriefs with stakeholders. Post-session debriefs help researchers discuss emerging findings as well as get the bigger picture from the stakeholder team. As stakeholders are closer to the category, their interpretation of data can provide crucial directions for report writing.
“It almost matters more - what did the stakeholder team take out of what we've heard and learned because they know the bigger picture, they know what they've looked at and learned before.”
-Janet Standen
The usage of AI and technology is fascinating. With the advent of new AI tools, there seems to be a push towards faster and faster report writing and shorter timelines, so we asked researchers...
How do they leverage AI tools? And how might AI tools help them balance speed and rigor?
Researchers are proactively trying to learn how AI can help them do their job better. Researchers are happy (and possibly) eager to upskill. One of the researchers we met had not tried any specific AI tools but he did dabble in experimenting with ChatGPT for several qualitative research processes including Discussion guide writing.
I've started using ChatGPT to do analysis. And I had spent months watching webinars and listening to the talks and reading articles before I had even touched it. And I was really unsure of how it would work. And then I did a project where I used it to write the discussion guide. I had samples of discussion guides, but I found a great article that had talked about prompts you could use to write a discussion guide. And so I included questions and a structure, and it produced something that was a good starting place.
-Doug Keith
Interesting Doug found ways to utilize ChatGPT for asynchronous as well as synchronous data.
This time the output was in spreadsheet form - the responses to all the questions. Then, I took the key questions. There were 19 respondents. For the key questions, I would take each question and I would put it in ChatGPT and say, imagine you're a market researcher and you have to do this analysis and tell me what the basic themes are for this data - and I would cut and paste the responses in Word. I ended up using all that material for the report. It was actually too much material, but kind of in a good way.
-Doug Keith
In general, researchers leverage AI tools for two major parts of the analysis workflow:
- Automated transcription
- Automated Analysis
Naturally, we wanted to learn how good the technology is at both the processes.
Automatic Transcription: While researchers largely feel automatic transcription is still not totally accurate. Yet, at least for the English language it has a level of quality where most researchers can largely rely on it. Some researchers get automated transcription and do the necessary correction themselves. Some researchers still get manual transcription done but in note taking format. Also, automatic transcription might be used in combination with note takers.
Researchers were happy with their transcription quality especially with CoLoop and Reduct.
Ability to edit the transcription
CoLoop did not provide the ability to edit the transcription which can be annoying especially if AI makes consistent mistake with terminology.
The problem is that you cannot, at-the-moment, amend. So for example, I was doing a project for home insurance and whenever people, even English consumers, when they were saying cover, it was transcribed as COVID. Regardless of who's talking, I had consumers from Scotland and consumers from Wales and consumers from England. Regardless of the accent. If I could just amend, I could just say, this is not “cover,” this is “COVID,”,
-Leonore Saintville
Speaker Identification
AI transcription is not necessarily good at identifying speakers. It might confuse between moderator and respondent.
The other thing that's always big for us is how does AI handle identifying what the moderator says versus what the participant says. Like it needs to be able to identify what speaker is the moderator and take that out of the analysis.
-Janet Standen
Automated Analysis
There is some scepticism around AI’s ability to do good quality analysis. While there is a spectrum of researchers who haven’t tried anything yet through to researchers who have tried automated analysis as recently as their last project. Overall, the consensus is that AI tools might provide a good starting point.
Let’s briefly evaluate common pros and cons of using AI tools
Pros of using AI tools
AI is a good starting point:
Using AI generated summaries can help researchers develop the initial picture in a more through fashion rather than just relying on memory. While AI is unlikely to throw any nuanced insights, a good AI tool can lay down bigger themes. With researcher’s involvement and context understanding, AI analysis can be a great starting point.
So, If I was using an AI tool like CoLoop, where I don't have to worry about the budget, I will put everything in. I will ask it to summarize each unit separately. Then, I might ask it to summarize different segments like these 3 non users groups vs summarize these 3 IDIs from Fashion Industry. And I think then I will summarize each of the summaries on my own. I am still reading those AI generated summaries but having moderated the sessions and then reading those AI generated summaries, it's going to prompt me to think about things that are not top of mind after three days of moderating. It's just helping me bring all the points together and make sure I didn't miss anything (While creating the final report story).
-Jenny Brandt
More thorough analysis compared to notes: The usage of AI did not necessarily save time but on the contrary AI helped in building a more thorough topline. If the researcher was relying primarily on notes while giving topline, the use of AI helped in picking up nuances that might have been missed in the notes.
The client asked for a topline based on my notes (at the last minute) and I arranged my notes by the subject area in the deck. But then I realized how much skimpy the notes were but when I did this process with ChatGPT, it gave me significantly more information.
-Doug Keith
Reduction of cognitive load: AI might help in doing the busy work involved in report writing process more efficiently. For example, finding supporting verbatim.
I think that cognitive resources it may have saved, in the sense that I didn't have to rack my brain to try and remember what was said, especially that I felt like there was some level of detail I might be missing.
-Doug Keith
Cons of using AI tool
Researchers also experienced some limitations while trying out AI tools.
Gives confused Analysis when concepts are used
AI tools can give good high-level summaries/themes when using for the exploratory research but may confuse one concept for another for the developmental research. AI tools struggle with complex projects that involve a lot of stimuli/concepts. That’s primarily because AI starts mixing up the data between concepts. Also, researchers can’t rely on automated transcription if they haven’t been part of the process (Moderated or heard the interview through an interpreter.)
Some of the tools like Co-loop does provide functionalities to code the concepts. Although this can quickly become cumbersome if the study involves too many concepts.
For example, I was doing research for a human insurance company, and they had lots of stimuli, lots and lots of stimuli. The artificial intelligence can't help you when you have lots of stimuli. That's really a gap. I can have a transcript, but then I must go back inside the artificial intelligence tool to code. I had claims, picture, effort for advertising, etc. So, it would be very messy, and I had to rotate all the material. So, then it becomes very difficult for the artificial intelligence to do anything.
-Leonore Saintville
High decibel claims of providing insights
Researchers mentioned some platforms that claims to create directly utilizable qualitative presentations. (One of such tools mentioned was Ezythemes) While the claims are intriguing, they are far from true regarding the delivery. AI is good at summarising the data. However, it is unable to produce any nuanced insight that human research analysis can. The key reason is it lacks the years of human experience that the researchers might have gathered in the domain. (Plus, the years of evolutionary perspectives that we have gained as human being)
I think they're good at basically summarizing well. What’s been said but into themes. They're not quite insights. This is where I think AI is at right now. As a researcher you need to really dig deep and understand. I don't think AI is ever going to get there or it's going to be a long time from now. What's missing is say I'm working on a report for let’s say an automotive client- I have 15 years of automotive experience and in that sense, AI is lacking this type of context and understanding.
-Dean Stephans
Hallucination
Some of the AI tool hallucinates and provides data that is not part of the study transcripts and this can be pretty scary. While researchers did not mention specific examples. They did experience this with some of the tools. Again, this is solvable to a good extent by giving the right instructions. However, researchers did mention spending significant amount of time to verify data quality.
Ethical considerations while using AI tools.
Did researchers have any concerns before using AI tools?
You Bet. The first and foremost concern remains the fear of LLMs using sensitive and proprietary study data for LLM training. Can you trust ChatGPT with the sensitive study data?
Well, turns out you can if you are using a paid plan or APIs. Sam Altman has categorically assured everyone around this multiple times. Most AI tools do not use the paid data for training. This was a big concern amongst all researchers and each verified the safety of proprietary data before using AI tools.
Conclusion
So, how should researchers leverage AI tools?
All in all, the landscape of qualitative research analysis is changing fascinatingly fast. The need for faster reporting process is becoming mainstream. The AI tools seem useful and hence are here to stay. However, as a researcher, you need to understand the limitations of AI tools and ensure you leverage them responsibly. At the current state of AI ‘Filling in the gaps’ is where the AI tools can truly shine. We are many ages away from ‘Insights at the click of a button’. Use AI tools as a starting point. AI tools can help in identifying the large themes emerging from the data. With the contextual understanding that you have as researcher, you can achieve best of both the worlds in terms of rigor and speed.
About US:
flowres is a platform built by myMRPlace. flowresAI is an online qualitative research tool built on our proprietary AI technology. Like other AI tools our AI is also improving and not perfect yet. We provide human transcription and human analysis over the top of AI abilities through myTranscriptionPlace (our language localization service). We are solution-focused and believe in ensuring the accurate data at speed by combining best of technology and human insight. flowres AI was tested by two of the researchers mentioned in this article, and they had raving reviews about the utility of the tool. One of the researchers evaluated flowres along with 5 other AI tools and flowres did fair better than all the other five AI tools. To know more about flowres, please feel free to reach out to me at [email protected]
*Harry Potter 1 and 2 together make 162085 words (a 60-minute Interview produces anywhere between 8 to 12K words (this is correlated with our internal data but of course this can vary hugely depending on format). At just 8K words per interview, 162085 words translate to a total of 20.26 Interviews.
